{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using FAI to solve Atari environments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TLDR \n",
    "\n",
    "1. In the notebook toolbar click Kernel -> **Restart & Run all**.\n",
    "2. **Wait a bit while you enjoy how the Agent plays MsPacman**.\n",
    "3. **You should have finished** at least the **first level of MsPacman-v0** using a uniform prior, and about 150 samples per action.\n",
    "4. There is a **video** of the game played **inside** the ***videos* folder** of this repository."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import everything we will need"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fractalai.model import RandomDiscreteModel\n",
    "from fractalai.environment import ExternalProcess, ParallelEnvironment, AtariEnvironment\n",
    "from fractalai.fractalmc import FractalMC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Available games "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a list of all the Atari games that can be played in Openai Gym using RGB images as observations. Just by changing the game name you can see how the algorithm performs on different environments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**['AirRaid-v0',  'Alien-v0', 'Amidar-v0', 'Assault-v0', 'Asterix-v0', 'Asteroids-v0', 'Atlantis-v0', 'BankHeist-v0',\n",
    " 'BattleZone-v0', 'BeamRider-v0', 'Berzerk-v0', 'Bowling-v0', 'Boxing-v0', 'Breakout-v0', 'Carnival-v0',\n",
    " 'Centipede-v0', 'ChopperCommand-v0', 'CrazyClimber-v0', 'DemonAttack-v0', 'DoubleDunk-v0', 'ElevatorAction-v0',\n",
    " 'Enduro-v0', 'FishingDerby-v0', 'Freeway-v0', 'Frostbite-v0', 'Gopher-v0', 'Gravitar-v0', 'Hero-v0', 'IceHockey-v0', 'Jamesbond-v0', 'JourneyEscape-v0', 'Kangaroo-v0', 'Krull-v0', 'KungFuMaster-v0', 'MontezumaRevenge-v0', 'MsPacman-v0', 'NameThisGame-v0', 'Phoenix-v0', 'Pitfall-v0', 'Pong-v0', 'Pooyan-v0', 'PrivateEye-v0', 'Qbert-v0',\n",
    " 'Riverraid-v0', 'RoadRunner-v0', 'Robotank-v0', 'Seaquest-v0', 'Skiing-v0', 'Solaris-v0', 'SpaceInvaders-v0',\n",
    " 'StarGunner-v0', 'Tennis-v0', 'TimePilot-v0', 'Tutankham-v0', 'UpNDown-v0', 'Venture-v0', 'VideoPinball-v0',\n",
    " 'WizardOfWor-v0', 'YarsRevenge-v0', 'Zaxxon-v0']**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Ram as observations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of a matrix of pixels, you can also use the ram of the Atari as observations. This will make the calculations a bit lighter, so do not be afraid to check it out!\n",
    "\n",
    "In order to use RAM as observations, add the \"-ram-\" suffix after the game name, and before \"v0\", as shown here:\n",
    "\n",
    "> 'MsPacman-v0' --> 'MsPacman**-ram**-v0'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpreting the parameter choice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The agent relies on four parameters:\n",
    "\n",
    "- **Fixed steps**: It is the number of consecutive times that we will apply an action to the environment when we perturb it choosing an action. Although this parameter actually depends on the Environment, we can use it to manually set the frequency at which the Agent will play. Taking more consecutive actions allows for exploring further in the future at the cost of less reaction time.\n",
    "\n",
    "- **Time Horizon**: This value represents \"how far we need to look into the future when taking an action\". A useful rule of thumb is **Time Horiozon = Nt / Fixed steps**, where **Nt** is the number of frames that it takes the agent to loose one life, (die) since the moment it performs the actions that inevitably lead to its death. This parameters determines the time horizon of the bigger potential well that the Agent should be able to escape.\n",
    "\n",
    "- **Max states**: This is the maximum number of walkers that can be part of the Swarm. This number is related to \"how thick\" we want the resulting causal cone to be. The algorithm will try to use the maximum number of walkers possible. \n",
    "\n",
    "- **Max samples**: This is the maximum number of times that we can make a perturbation when using a Swarm to build a causal cone. It is a superior bound, and the algorithm will try to use less samples to meet the defined **time horizon**. It is a nice way to limit how fast you need to take an action. A reasonable value could be **max walkers** \\* **time horizon** \\* ***N***, being ***N=5*** a number that works well in Atari games, but it depends on the task.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can take a look at the [Fractal AI Performance Sheet](https://docs.google.com/spreadsheets/d/1JcNw2L0YL_I2iGZPJ0bNKJshlTaqMuEl5CP2W5zie6M/edit?usp=sharing) to check the parameters we used to run our experiments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practical example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ### Minimal Pacman"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will tune the Agent to get a decent score on MsPacman using the minimum amount of computational resources possible. We will deliberately set a very small amount of computational resources for calculating an action.\n",
    "\n",
    "Doing that we want to address concerns about edge cases of the theory, by showing how the algorithm performs when the size of the swarm Swarm is very little with respect to the size of the state space.\n",
    "\n",
    "In order to do so, we can give the parameters the following values:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Environment Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "name = \"MsPacman-ram-v0\"\n",
    "render = True # It is funnier if the game is displayed on the screen\n",
    "clone_seeds = True  # This will speed things up a bit\n",
    "max_steps = 1e6  # Play until the game is finished.\n",
    "n_repeat_action = 1  # Atari games run at 20 fps, so taking 4 actions per seconds is more \n",
    "reward_limit = 20000\n",
    "render_every = 2\n",
    "dt_mean = 3\n",
    "dt_std = 2\n",
    "min_dt = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### FAI parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_samples = 6000  # Let see how well it can perform using at most 300 samples per step\n",
    "max_walkers = 100 # Let's set a really small number to make everthing faster\n",
    "time_horizon = 30  # 50 frames should be enough to realise you have been eaten by a ghost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With these parameters we are aiming for 100 samples per step, saving up to another 200 samples in case the agent runs into trouble. Using such a low number of samples will mean that the performance could vary widely among different runs.\n",
    "\n",
    "In our tests, this agent was capable of finishing the first level most of the times if we set max_states = 15 (150 samples). Using only 100 samples will make it hard for the Agent to find rewards that are far away, so at the end of the first level you will be relying mostly on luck.\n",
    "\n",
    "If you want to get better scores, just increase the values of the parameters accordingly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env = ParallelEnvironment(name=name,env_class=AtariEnvironment,\n",
    "                          blocking=False, n_workers=8, n_repeat_action=n_repeat_action)  # We will play an Atari game\n",
    "model = RandomDiscreteModel(max_wakers=max_walkers,\n",
    "                            n_actions=env.n_actions) # The Agent will take discrete actions at random\n",
    "\n",
    "fmc = FractalMC(model=model, env=env, n_walkers=max_walkers,\n",
    "                reward_limit=reward_limit, render_every=render_every,\n",
    "                time_horizon=time_horizon, dt_mean=dt_mean, dt_std=dt_std, accumulate_rewards=True, min_dt=min_dt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fmc.run_agent(render=True, print_swarm=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Replay Game"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fmc.render_game(sleep=1/40)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We will really appreciate your feedback"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
